[https://nvbugs/6490028][fix] Bump only the cross-library `cublas_tolerance` from 1.05 to 1.10 in the test… by trtllm-agent · Pull Request #16788 · NVIDIA/TensorRT-LLM

trtllm-agent · 2026-07-23T07:48:37Z

Summary

Root cause: cublas_tolerance=1.05 on a cross-library ~28us GEMM comparison profiled over only 20 iterations is inside the measurement's own run-to-run noise floor, so the test flakes above threshold by <1% intermittently.
Fix: Bump only the cross-library cublas_tolerance from 1.05 to 1.10 in the test; keep the intra-CuteDSL 1.05 tolerance unchanged. Added a comment explaining why the two tolerances differ.
Automated fix generated by repair-bot

Test plan

Verify fix on the same GPU type as the original failure
Check for regressions in related tests

Links

Bug: https://nvbugs/6490028

Dev Engineer Review

In test_cutedsl_nvfp4_heuristic_matches_full_sweep, loosened the cuBLASLt cross-library comparison bound by changing cublas_tolerance from 1.05 to 1.10.
Kept the intra-CuteDSL tolerance at 1.05, with added comments documenting that the change is specifically for the CuteDSL-vs-cuBLASLt comparison and is intended to tolerate run-to-run timing noise.
Change is narrowly scoped to the heuristic-test tolerance/expectation; no API/config/error-handling behavior changes.

QA Engineer Review

Modified test: tests/unittest/_torch/misc/test_autotuner.py::test_cutedsl_nvfp4_heuristic_matches_full_sweep (updated cross-library tolerance and rationale comments).
Test-list change: updated tests/integration/test_lists/waives.txt by removing the waiver/skip entry for unittest/_torch/misc/test_autotuner.py::test_cutedsl_nvfp4_heuristic_matches_full_sweep.
Coverage: not identified as present/updated in test-db/ or qa/.
Verdict: needs follow-up.

coderabbitai · 2026-07-23T07:51:44Z

Walkthrough

The CuteDSL autotuner test increases the cuBLASLt comparison tolerance from 1.05 to 1.10, documents cross-library profiling variability, and removes the test from the integration waivers list.

Changes

Autotuner tolerance adjustment

Layer / File(s)	Summary
Cross-library tolerance validation `tests/unittest/_torch/misc/test_autotuner.py`, `tests/integration/test_lists/waives.txt`	The CuteDSL-versus-cuBLASLt assertion uses a 1.10 tolerance with comments documenting profiling jitter, and the test is removed from the waivers list.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Suggested reviewers: bowenfu

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is specific and accurately summarizes the main change: increasing the cross-library cublas_tolerance for the test.
Description check	✅ Passed	The description covers the issue, fix, test plan, and bug link, but it doesn't follow the template's exact section headings.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/unittest/_torch/misc/test_autotuner.py`:
- Around line 1263-1267: Register
test_cutedsl_nvfp4_heuristic_matches_full_sweep in both the test-db and qa
test-list registries under tests/integration/test_lists, and add the
corresponding CBTS coverage entry using the repository’s existing conventions so
the test is included in CI and QA routing.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f9ea4ac7-e591-44a6-be38-71ef96ceaa46

📥 Commits

Reviewing files that changed from the base of the PR and between 93efd33 and 75e2cee.

📒 Files selected for processing (1)

tests/unittest/_torch/misc/test_autotuner.py

… NVFP4 test The heuristic-vs-cuBLASLt comparison in test_cutedsl_nvfp4_heuristic_matches_full_sweep uses a 1.05x tolerance on a ~28us GEMM kernel profiled over only 20 iterations. Run-to-run jitter from DVFS, L2 residency, and interleaving with cuBLAS autotune warmup easily exceeds 5% at this kernel scale, so the bound is inside the measurement noise floor and the test flakes above threshold by <1% intermittently. Relax cublas_tolerance to 1.10 (the intra-CuteDSL sweep-vs-heuristic 1.05 tolerance is unchanged -- that comparison is the actual pruning-feature invariant and is not cross-library). Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>

Signed-off-by: handongl <handongl@nvidia.com>

coderabbitai

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

tests/integration/test_lists/waives.txt (1)
1-1: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add the required NVIDIA copyright header.

This modified file has no visible NVIDIA copyright header. Add the header and update its copyright year as required by the repository guidelines.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/test_lists/waives.txt` at line 1, Add the
repository-standard NVIDIA copyright header at the beginning of
tests/integration/test_lists/waives.txt, using the current required copyright
year while preserving the existing waiver entry unchanged.
Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@tests/integration/test_lists/waives.txt`:
- Line 1: Add the repository-standard NVIDIA copyright header at the beginning
of tests/integration/test_lists/waives.txt, using the current required copyright
year while preserving the existing waiver entry unchanged.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 9d9ad966-e983-4fb9-a46c-86be2b10f4f7

📥 Commits

Reviewing files that changed from the base of the PR and between 75e2cee and 74598b7.

📒 Files selected for processing (2)

tests/integration/test_lists/waives.txt
tests/unittest/_torch/misc/test_autotuner.py

🚧 Files skipped from review as they are similar to previous changes (1)

tests/unittest/_torch/misc/test_autotuner.py

peaceh-nv · 2026-07-24T09:37:46Z

/bot run

tensorrt-cicd · 2026-07-24T09:44:03Z

PR_Github #61558 [ run ] triggered by Bot. Commit: 74598b7 Link to invocation

tensorrt-cicd · 2026-07-24T10:35:22Z

PR_Github #61558 [ run ] completed with state SUCCESS. Commit: 74598b7
/LLM/main/L0_MergeRequest_PR pipeline #49769 completed with status: 'SUCCESS'

CI Report

Link to invocation

trtllm-agent requested a review from a team as a code owner July 23, 2026 07:48

trtllm-agent assigned peaceh-nv Jul 23, 2026

trtllm-agent requested review from cascade812 and tongyuantongyu July 23, 2026 07:48

github-actions Bot assigned trtllm-agent Jul 23, 2026

coderabbitai Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread tests/unittest/_torch/misc/test_autotuner.py

trtllm-agent and others added 2 commits July 24, 2026 00:26

[nvbugs/6490028][chore] Remove stale waiver after fix

74598b7

Signed-off-by: handongl <handongl@nvidia.com>

trtllm-agent force-pushed the repair-bot-bug6490028 branch from 75e2cee to 74598b7 Compare July 24, 2026 07:27

trtllm-agent requested review from a team as code owners July 24, 2026 07:27

trtllm-agent requested review from BowenFu, EmmaQiaoCh, QiJune, StanleySun639, tburt-nv and xinhe-nv July 24, 2026 07:27

coderabbitai Bot reviewed Jul 24, 2026

View reviewed changes

peaceh-nv approved these changes Jul 24, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[https://nvbugs/6490028][fix] Bump only the cross-library `cublas_tolerance` from 1.05 to 1.10 in the test…#16788

[https://nvbugs/6490028][fix] Bump only the cross-library `cublas_tolerance` from 1.05 to 1.10 in the test…#16788
trtllm-agent wants to merge 2 commits into
NVIDIA:mainfrom
tensorrt-cicd:repair-bot-bug6490028

trtllm-agent commented Jul 23, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jul 23, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Uh oh!

peaceh-nv commented Jul 24, 2026

Uh oh!

tensorrt-cicd commented Jul 24, 2026

Uh oh!

tensorrt-cicd commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

trtllm-agent commented Jul 23, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Links

Uh oh!

coderabbitai Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

peaceh-nv commented Jul 24, 2026

Uh oh!

tensorrt-cicd commented Jul 24, 2026

Uh oh!

tensorrt-cicd commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

trtllm-agent commented Jul 23, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 23, 2026 •

edited

Loading